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AIR  TRAFFIC  CONTROL  SPECIALISTS  IN  THE 
AIRWAY  SCIENCE  CURRICULUM  DEMONSTRATION  PROJECT  1984-1990: 
THIRD  SUMMATIVE  EVALUATION 


INTRODUCTION 

Purpose  of  Airway  Science 
Curriculum  Demonstration  Project 

The  purpose  of  the  Airway  Science 
Curriculum  Demonstration  Project  (ASCDP)  was 
to  compare  the  performance,  Job  attitudes,  and 
perceived  supervisory  potential  of  individuals 
recruited  for  4  of  the  FAA’s  technical 
occupations  who  have  an  aviation-related, 
college-level  education,  or  its  equivalent,  with 
individuals  recruited  for  the  same  occupation 
through  traditional  methods  (Office  of  Personnel 
Management  (0PM),  1983a,  1983b).  The 

ASCDP  represented  an  alternative  to  the 
traditional  hiring  and  recruitment  strategy  used 
by  the  FAA.  This  report  presents  a  summative 
evaluation  of  the  project  as  of  the  end  of 
calendar  year  1990. 

Objectives  of  the  Project 

Evaluation  research,  such  as  described  in 
this  report,  is  concerned  with  securing  evidence 
on  the  attainment  of  program  objectives  in  terms 
of  operationally  defined  criteria  (Bloom,  1967; 
Rossi  &  Freeman,  1985).  The  Federal  Register 
announcements  of  the  ASCDP  (0PM,  1983a, 
1983b)  specified  4  broad  areas  for  evaluation. 
The  first  area  focused  on  the  recruitment  and 
hiring  of  individuals  who  have  completed,  or 
have  the  equivalent  of,  the  model  curriculum. 
Data  on  the  rates  of  application  to,  certification 
for,  and  hiring  from  the  Airway  Science  registers 
have  been  previously  reported  (Broach,  1990). 
The  results  of  that  second  summative  evaluation 
indicated  that  hiring  under  the  ASCDP 
consistently  lagged  behind  both  the  goals  set 
forth  in  the  Federal  Register  announcements  and 
the  annually  updated  agency  goals.  Second,  die 
concept  that  individuals  with  this  Airway  Science 
background  were  better  able  to  perform  the  Job 
than  individuals  recruited  by  existing  methods 
was  to  be  evaluated.  The  third  evaluative  topic 
identified  by  the  Federal  Register  encompassed 
Job  performance,  attitudes,  and  supervisory 
potential.  Finally,  the  Federal  Register  required 
determination  of  the  impact  of  the  program  "on 
the  employment  and  career  progression  of 
women  and  minority  candidates"  (0PM,  1983,  p. 
11673). 


Project  Background 

The  inception,  implementation,  and  revision 
of  the  ASCDP  have  been  previously  described  in 
reports  by  Clough  (1985,  1986a,  1986b,  1986c, 
1987a,  1987b,  1988)  and  Broach  (1990).  The 
Airway  Science  Curriculum  was  proposed  by 
former  FAA  Administrator  J.  Lynn  Helms  in 
1981;  the  University  Aviation  Association 
(UAA),  a  professional  organization  for  non¬ 
engineering  collegiate  aviation  educators  and 
institutions,  provided  assistance  in  refining  the 
general  academic  concepts  into  a  specific  college 
curriculum.  The  FAA  submitted  a  demonstration 
project  evaluation  plan  to  0PM  in  1983,  which 
was  approved  (0PM  1983a,  1983b).  Originally 
plann^  as  a  5-year  demonstration  program, 
0PM  granted  a  4-year  extension  to  the 
demonstration  project  in  1987,  when  initial  FAA 
hiring  rates  were  welt  below  projections  (Broach, 
1990).  Rigidities  in  program  requirements  posed 
difficulties  for  participating  institutions,  as  well, 
(Bowen,  1990)  that  were  remedied  by  program 
revisions  implemented  in  1989  by  the  FAA  and 
UAA.  A  summative  evaluation  was  conducted 
jointly  by  the  FAA  Office  of  Training  and 
Higher  Education  (AHT),  the  Civil  Aeromedical 
Institute  (CAMI),  and  the  UAA  in  1990  for  the 
Office  of  the  Secretary  of  Transportation.  The 
qualitative  evaluation  by  the  Higher  Education 
and  Advanced  Technology  Staff  (AHT-30;  1990) 
addressed  program  management,  lessons  learned, 
and  recommendations.  That  report  concluded  that 
although  the  ASCDP  had  not  fully  met  its  stated 
objectives  as  of  calendar  year  1989,  the 
identifiable,  institutional  and  organizational 
benefits  outweighed  that  apparent  lack  of  success. 
The  CAMI  staff  evaluation  ^roach,  1990) 
provided  quantitative  data  relative  to  program 
objectives  within  the  evaluation  framework 
stipulated  by  the  Federal  Register  announcement 
of  the  demonstration  project.  That  report 
concluded  that  the  ASCDP  had  been  less  foan 
successful  in  achieving  its  stated  program 
objectives.  The  UAA  evaluation  (UAA,  1990) 
focused  on  the  experience  of  the  participating 
schools.  The  UAA  analysis  indicated  that  the 
ASCDP  had  been  successful  in  providing  entry- 
level  technical  and  managerial  employees  to  the 
broader  aviation  industry  rather  than  the  FAA. 
The  UAA  evaluation  also  noted  positive  effects 
on  institutions  of  higher  education  stemming 
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and  (c)  aviation  safety  inspectors.  The  population 
mean  age  at  the  time  of  hire  was  27  {SD  =  5,  N 
=  10,228),  and  ranged  from  18  to  58  years  of 
age.  Of  the  10,229  cases  indicating  gender, 
15.8%  (N  =  1,623)  were  female.  Minorities 
account^  for  9.1%  (N  =  939)  of  the  10,141 
cases  providing  ethnic  group  identification  data. 
Non-veterans  representwl  88.2%  {N  =  9,065)  of 
the  10,185  cases  for  which  veteran’s  preference 
information  was  available.  The  modal  Vocational 
level  for  10,226  cases  was  the  equivalent  of  1  or 
2  years  of  post-secondary  education  (43.4%,  or 
N  =  4,463),  either  in  technical/  trade  schools  or 
formal  academic  institutions.  Just  0.2%  {N  =  18) 
had  not  completed  high  school  or  obtained  an 
equivalency  diploma;  some  20.3%  {N  =  2,085) 
had  completed  high  school  or  its  equiv^ent. 
Some  7.7%  of  the  population  (N  =■  793)  had 
completed  3  or  4  years  of  college.  Just  1.4%  {N 
=  148)  had  completed  a  post-baccalaureate 
degree.  A  total  of  26.5%  (N  =  2,719)  had 
completed  a  bachelor’s  degree. 


from  the  ASCDP.  Based  on  the  AHT  and  UAA 
evaluations  in  particular,  0PM  agreed  to 
terminate  the  ASCDP  as  a  formal  demonstration 
in  1990.  At  that  point.  Airway  Science  became 
an  institutionalizV  permanent  FAA  program 
administered  by  the  Office  of  Training  and 
Higher  Education  (Director,  Office  of  Training 
and  Higher  Education,  1990). 

Evaluation  Design 

The  evaluation  design  for  the  ASCDP  was 
established  in  the  1983  Federal  Register 
announcement.  However,  as  noted  by  both 
Clough  (1986a, b)  and  Broach  (1990),  the 
evaluation  design  confounded  the  influence  of  the 
specific  curriculum  on  occupational  success  with 
the  impact  of  an  alternative  selection  strategy. 
The  design  required  comparison  of  outcomes  for 
the  persons  hired  via  the  ASCDP  registers  with 
outcomes  for  a  non-equivalent  control  group  of 
persons  hired  via  the  "traditional"  competitive 
0PM  registers.  The  original  research 
methodology  assumed  a  clearcut  distinction 
between  Airway  Science  and  traditional  hires. 
However,  the  Airway  Science  rating  guides 
provided  for  substitutions  of  relevant  experience 
for  formal  education;  as  a  result,  persons  hired 
via  the  special  Airway  Science  registers  included 
both  graduates  from  approved  Airway  Science 
programs  as  well  as  non-college  applicants  that 
qualified  by  reason  of  experience  and  education. 
As  a  result,  the  research  design  confounded 
hiring  strategy  with  the  Airway  Science 
curriculum.  Such  a  confounded  ex  post  facto 
design  is  essentially  uninterpretable  (Cook  & 
Campbell,  1976).  As  a  consequence,  causal 
interpretations  about  the  effects  of  the  curriculum 
and  hiring  strategy  cannot  be  made  on  the  basis 
of  evaluation  results.  However,  the  ASCDP  can 
be  qualitatively  assessed  in  relation  to  the  broad 
program  objectives  defined  for  the  project.  This 
third  summative  evaluation,  therefore,  focuses  on 
the  degree  to  which  the  ASCDP  achieved  its 
agency-related  program  objectives. 

METHOD 

Sample 

Population  characteristics 

Overall.  Data  on  a  total  (N)  of  10,277 
cases  from  3  technical  occupations  were  available 
from  the  Airway  Science  data  base;  (a)  air  traffic 
control  specialists;  (b)  electronics  technicians; 


By  occupation.  Air  traffic  control  specialists 
(ATCS)  represented  94.3%  (N  =  9,693)  of  the 
population  in  the  Airway  Science  data  base. 
Hires  from  the  Air  Traffic  Airway  Science 
register  represented  only  3.2%  (N  =  312)  of  the 
controllers.  There  were  only  47  graduates  from 
recognized  schools  with  degrees  in  an  approved 
area  of  concentration  among  the  controllers  hired 
between  1984  and  1990.  Electronic  technician*: 
(ET)  represented  a  mere  2.2%  (IV  =  228)  of  that 
population;  aviation  safety  inspectors  (ASl) 
accounted  for  3.4%  (N  =  355)  of  the  cases  in 
the  Airway  Science  data  base.  Hires  from  the  ET 
Airway  Science  register  accounted  for  48.9%  (A 
=  112)  of  technician  new  hires.  "Hiere  were  no 
graduates  from  recognized  schoc's  with  degrees 
in  an  approved  area  of  concentration  in  this 
sample  of  technicians  hire^  between  1984  and 
1990.  Hires  from  the  .\j»I  Airway  Science 
register  represented  12  4%  (N  =  44)  of  the 
inspectors  hired  between  1984  and  1990;  just  6 
of  these  Airway  Science  hires  were  graduates  of 
recognized  colleges  with  a  degree  in  an  approved 
area  of  concentration.  Demographic 
characteristics  of  the  ASCDP  population  are 
presented  by  occupation  in  Table  1 . 


Because  controllers  constituted  the  bulk  of 
the  ASCDP  population  and  accounted  for  most  of 
the  hiring  under  the  terms  of  the  ASCDP,  this 
evaluation  focused  on  the  experience  of  the 


Evaluution  sample 
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Table  1 


Demographic  Characteristics  of  Population  by  Occupation 


Characteristic 

Category 

ATCS 

ET 

ASI 

Age 

M 

26.4 

34.4 

42.4 

SD 

3.4 

8.3 

7.8 

Sex 

Male 

83.2%  (8,062) 

89.5%  (205) 

95.5%  (339) 

Female 

16.4%  (1,590) 

7.4%  (  17) 

4.5%  (  16) 

Missing 

0.4%  (  41) 

3.1%  (  7) 

Race 

Nonminority 

89.7%  (8,696) 

80.3%  (184) 

90.7%  (322) 

Minority 

9.0%  (  875) 

14.8%  (34) 

8.5%  (30) 

Missing 

1.3%  (  122) 

4.8%  (  11) 

0.8%  (  3) 

Veteran 

Non-veteran 

91.8%  (8,901) 

41.5%  (95) 

19.4%  (69) 

Veteran 

7.6%  (  735) 

50.2%  (115) 

76.1%  (270) 

Missing 

0.6%  (  57) 

8.3%  (  19) 

4.5%  (  16) 

Education 

LT  High  school 

0.2%  (  18) 

High  school 

20.4%  (1,977) 

16.2%  (37) 

20.0%  (71) 

2-year  college 

43.3%  (4,196) 

58.5%  (134) 

37.5%  (133) 

3-4  yrs  college 

7.8%  (  759) 

4.4%  (  10) 

6.8%  (24) 

Bachelor  degree 

26.6%  (2,581) 

15.7%  (36) 

28.7%  (102) 

Postgraduate 

1.2%  (  119) 

2.2%  (  5) 

6.8%  (24) 

Missing 

0.4%  (  43) 

3.1%  (  7) 

0.3%  (  1) 

Airway  Science 

None 

96.8%  (9,379) 

51.1%  (117) 

88.9%  (311) 

Education 

Experience  only 

0.3%  (  28) 

7.4%  (  17) 

1.7%  (  6) 

Non-avtn  major 

0.5%  (  51) 

16.2%  (37) 

3.1%  (  11) 

Aviation  major 

0.4%  (  39) 

0.9%  (  2) 

0.8%  (  3) 

Avtn  graduate 

1.5%  (  147) 

0.9%  (  2) 

3.7%  (  13) 

Airway  Science 

0.5%  (  47) 

1.7%  (  6) 

Missing 

0.0%  (  2) 

23.6%  (54) 

1.4%  (  4) 

Note:  Ni!rr.ber  in  category  shown  in  parentheses  to  right  of  percentages 


ATCS  occupation  in  the  demonstration  project. 
Their  experience  as  an  occupational  group  can  be 
considered  representative  of  the  success  or  failure 
of  the  ASCDP.  Terminal  and  En  Route 
controllers  only  were  included  in  the  evaluation 
sample;  Flight  Service  Specialists  (FSS)  were 
excluded  because  of  the  fundamental  differences 
in  the  jobs  performed.  The  unequal  group  sizes 
between  traditional  hires  and  Airway  Science 
hires  in  the  ATCS  occupation  indicated  a 
requirement  for  sampling  the  traditional  hire 
population.  The  traditional  ATCS  hires  were 
stratified  by  education;  random  samples  were 
taken  from  each  stratum  such  that  a  number 
equal  to  Airway  Science  hires  with  the  same 
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educational  level  in  an  occupation  were  selected 
for  the  analysis.  The  demographic  characteristics 
of  the  evaluation  sample  of  controllers  are 
presented  in  Table  2.  Traditionally-hired 
controllers  were  slightly  older,  on  the  average  (M 
=  26.3,  SD  =  2.9  years),  than  controllers  hired 
via  the  Airway  Science  register  (A/  =  25.9,  SD 
~  3.0;  /(1, 310)  =  6.71,  p  j<  .05).  However, 
the  representation  of  women  in  the  2  groups  was 
not  significantly  different  (18.6%  of  traditional 
versus  13.8%  of  Airway  Science  hires,  Z  =  - 
1.63,  p  >  .05).  While  slightly  more  minority 
specialists  were  represented  in  the  Airway 
Science  group  (10.3%),  the  percentage  was  not 
significantly  different  from  their  representation  in 
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the  stratified  sample  of  traditionally-hired 
controllers  (6.1%,  Z  =  1.92,  p  >  .05).  Nor 
were  the  percentages  of  veterans  in  the  2  groups 
(5.7%  versus  7.7%)  significantly  different  (Z  =  - 
1.00,  p  >  .05).  Finally,  given  that  the  sampling 


frame  was  stratified  on  education,  there  was  no 
significant  difference  in  the  proportions  of 
undergraduate  college  graduates  (59.9%  versus 
58.7%)  between  the  2  samples  (Z  =  -0.31,  p  > 
.05). 


Table  2 


Demographic  Characteristics  of  Evaluation  Sample 


Characteristic 

Category 

Traditional 
(N  =  312) 

Airway  Science 
(N  =  312) 

Age 

M 

26.3 

25.9 

SD 

2.9 

3.0 

Sex 

Male 

81.4%  (254) 

73.1%  (228) 

Female 

18.6%  (  58) 

13.8%  ( 43) 

Missing 

13.1%  (41) 

Race 

Nonminority 

92.9%  (290) 

74.0%  (231) 

Minority 

6.1%  (  19) 

10.3%  (  32) 

Missing 

1.0%  (  3) 

15.7%  ( 49) 

Veteran 

Non-veteran 

94.2%  (294) 

92.3%  (250) 

Veteran 

5.7%  (  18) 

7.7%  (21) 

Missing 

13.1%  (41) 

Education 

HS  Diploma 

9.6%  (30) 

6.4%  ( 20) 

2-year  college 

19.9%  (  62) 

13.1%  (41) 

3-4  yrs  college 

6.7%  (  21) 

6.1%  (19) 

Bachelor  degree 

59.9%  (187) 

58.7%  (183) 

Postgraduate 

3.8%  (  12) 

2.6%  (  8) 

Missing 

13.1%  (41) 

Data  Collection  Instruments  and  Procedures 
General  framework 

Operational  criteria  defined  by  Clough  (1988) 
and  Broach  (1990)  for  this  evaluation  are 
presented  in  Table  3.  Data  collection  to  support 
the  evaluation  focused  on  (a)  identifying  new 
hires,  with  careful  attention  to  the  identification 
of  true  Airway  Science  graduates  entering  via 
the  traditional  0PM  registers,  (b)  extracting  and 
matching  required  data  from  relevant 
computerized  data  bases,  and  (c)  collecting 
altitudinal,  training  progress,  and  performance 
appraisal  data  in  annual  surveys.  Computerized 
data  sources  utilized  in  the  course  of  the 
evaluation  include  the  FAA  Consolidated 
Personnel  Management  Information  System 
(CPMIS)  and  the  CAMI  ATCS  Selection  data 


base.  Instruments  to  collect  altitudinal,  training, 
and  performance  data  were  designed  by  CAMI 
researchers  (Clough,  1987a,  1988).  Selected 
data  from  the  supporting  data  bases  and  from 
various  ASCDP  surveys  were  consolidated  into 
a  single  ASCDP  data  base,  as  described  ny 
Broach  (1990).  Participation  in  the  project  was 
a  condition  of  employment  for  Airway  Science 
hires.  In  contrast,  the  participation  of  the 
traditionally-hired  subjects  was  voluntary.  The 
participation  of  traditional  air  traffic  hires  was 
solicit^  at  the  time  of  entry  onto  duty  with  the 
agency  at  the  FAA  Academy.  Survey 
instruments  were  mailed  annually  to  participants 
and  their  supervisors,  based  on  the  enter-on-duty 
(EOD)  date.  Various  indices  and  scales  were 
constructed  to  operationally  represent  the  criteria 
for  evaluating  attainment  of  the  project 
objectives.  The  focus  of  this  evaluation  was  on 
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individual  outcomes  within  the  ATCS  occupation, 
such  as  remaining  with  the  FAA  and  technical 


competence,  rather  than  on  institutional 
outcomes,  such  as  ties  between  the  FAA  and 
academia. 


Table  3 

Project  Objectives  and  Evaluation  Criteria 


Project  Objectives 

Evaluation  Criteria 

Objective  2 A 

To  assess  the  unique  impact 
of  a  tailor-made  curriculum 
on  one’s  interest  in  pur¬ 
suing  an  aviation-related 
career. 

Aviation-related  education  completed  (semester  hours) 
Aviation-related  work  experience,  licenses,  and  certificates 
Rated  importance  of  aviation-related  career 

Objective  2B 

To  assess  the  unique  impact 
of  a  tailor-made  curriculum 
on  occupational  and  organi¬ 
zational  attrition. 

Occupational  attrition 

Organizational  attrition 

Objective  3 

To  improve  the  technical 
competence  of  the  FAA 
workforce. 

Occupational  training  performance 

Formal  performance  appraisal  results 

Objective  4 

To  develop  within  the  FAA 
an  increased  acceptance  of 
technological  change. 

Effect  of  automation  on  job  perceptions 

Perceived  changes  in  job  from  automation 

Objective  5 

To  recruit  individuals 
with  managerial  potential. 

Managerial  potential  as  rated  by  supervisor 

Objective  6 

To  attract  individuals  with 
greater  awareness  of  and 
skills  in  maintaining  positive 
human  relations. 

Interpersonal  skills  as  rated  by  supervisor 

Objective  7 

To  increase  female  and 
minority  representation. 

Proportions  female/minorities  retained 

Objective  8 

To  improve  employee 
perceptions  of  the  FAA. 

Degree  that  FAA  meets  employee  work  needs 

Job  satisfaction 
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RESULTS 

Interest  in  Aviation-Related  Career 
Amount  of  aviation-related  education 

As  might  be  expected.  Airway  Science  hires 
completed  significantly  more  aviation-related 
credit  hours  in  degree-oriented  programs  than 
members  of  the  traditional  hire  control  group. 
Specifically,  66.8%  of  the  Airway  Science 
controllers  reported  completing  30  or  more  credit 
hours,  while  79. 1  %  of  the  control  group  reported 
completing  less  than  6  aviation-related  credit 
hours  (x'(4)  =  228.9,  p  .001). 

Degree  of  aviation-related  work  experience 

Degree  of  aviation-related  work  experience 
was  represented  by  2  variables:  (a)  actual 
aviation  work  experiences  reported  by  subjects; 
and  (b)  the  number  of  aviation  certificates  or 
licenses  held  by  an  individual.  Just  7.7%  of  the 
Airway  Science  hires  reported  6  or  more 
aviation-related  work  experiences  such  as 
employment  as  a  pilot,  flight  instructor,  or 
mechanic;  40.1%  of  the  312  Airway  Science 
hires  reported  no  aviation-related  work 
experience  prior  to  employment  by  the  FAA.  In 
comparison.  Just  2.6%  of  the  control  group 
reported  6  or  more  aviation-related  work 
experiences,  while  44.2%  indicated  no  prior 
aviation  employment  before  Joining  the  agency. 
However,  while  the  overall  distribution  of 
responses  was  significantly  different  from  a 
chance  distribution  (x^(3)  =  10.0,  p  j<  .05),  the 
only  statistically  significant  difference  in  rates 
was  for  those  controllers  indicating  having  held 
6  or  more  aviation-related  Jobs.  A  significantly 
greater  percentage  of  controllers  hired  from  the 
Airway  Science  register  (7.7%)  reported  having 
held  6  or  more  aviation  Jobs  ^an  traditionally- 
hired  specialists  (2.6%;  Z  =  2.88,  p  _<  .01). 

Career  importance 

Controllers  from  both  groups  were  asked  to 
rate  the  importance  of  a  career  in  aviation  on  a 
5-point,  Likert-type  scale  with  the  following 
anchors;  1  =  Not  at  all  important;  2  =  Of 
limited  importance;  3  =  Of  moderate 
importance;  4  =  Of  considerable  importance;  5 
=  Of  very  great  importance.  As  described  by 
Broach  (1990),  these  data  were  collected  at  3 
points  in  time:  after  1  year  of  employment;  after 
2  years;  and  3  years  after  entering  on  duty  with 


the  FAA.  Airway  Science  hires  rated  the 
importance  of  a  career  in  aviation  as  more 
important  (Af  =  4.73)  after  1  year  of 

employment  than  traditionally-hired  controllers 
(M  =  4.45,  F(l,170)  =  7.74,  p  .01). 
However,  due  to  lower  response  rates  in  the 
second  and  third  years  of  data  collection,  as 
described  by  Broach  (1990),  those  apparent 
differences  disappeared.  After  2  years  of 
employment,  for  example,  the  rated  importance 
of  a  career  in  aviation  for  traditionally-hired 
controllers  (M  =  4.61)  was  not  statistically 
different  from  the  rating  given  by  Airway 

Science  controllers  (M  =  4.58;  F(l,35)  =  .02, 
p  >  .05).  A  similar  pattern  after  3  years  of 
employment  was  observed  =  4.71  vs. 

^AscDP  =  4.54,  F(l,16)  =  0.32,  p  >  .05). 

Overall,  a  career  in  aviation  appeared  to  be  more 
important  to  controllers  hired  from  the  Airway 
Science  register  than  for  specialists  hired  from 
the  traditional  competitive  register  in  the  first 
year  of  employment  only.  However,  the  small 
sample  sizes  and  resulting  lack  of  statistical 
power  do  not  support  reliable  inferences  about 
the  relative  importance  of  an  aviation  career  in 
subsequent  years  of  employment. 

Attrition 

For  purposes  of  this  evaluation  organizational 
attrition  was  defined  as  "retention  in  any 
occupation  or  loss  for  any  reason  of  the 
controller  by  the  FAA;"  that  is,  a  controller 
separated  or  resigning  from  the  agency  was  an 
organizational  loss.  Occupational  attrition  was 
defined  as  "separation  from  the  ATCS 
occupational  field  series  (GS-2152)  but  remaining 
in  the  FAA’s  employ.”  Overall,  the  rates  of 
organizational  attrition  for  controllers  in  the  2 
comparison  groups  from  1984  to  1990  were  not 
significantly  different  (Figure  1).  The  rate  of 
attrition  among  the  Airway  Science  hires  was 
41.7%,  white  that  in  the  sample  of  traditionally- 
hired  controllers  was  35.3%  (Z  =  1.64,  p  > 
.05).  On  one  hand,  the  organizational  loss  rate 
for  Airway  Science  controllers  was  no  different 
than  the  loss  rate  for  traditionally-hired 
specialists.  On  the  other  hand,  the  rate  of 
occupational  attrition  among  the  Airway  Science 
controllers  was  7.0%,  compared  to  a  rate  of  Just 
3.2%  in  the  sample  of  traditionally-hired 
specialists  (Z  =  2.16,  p  ^  .01).  Overall,  it 
appears  that  the  net  loss  rate  from  the  controller 
occupation  was  higher  among  the  Airway  Science 
hires  than  in  the  stratified  sample  of  traditionally- 
hired  air  traffic  control  specialists. 
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Figure  1 

Organizational  attrition  and  retention 
rates  by  comparison  groups 


Attrition  Retention 


I  ASCDP 


Traditional 


Technical  Competence 
Occupational  training 

The  critical  point  in  the  occupational  training 
of  controllers  is  the  FAA  Academy  ATCS  Screen 
(FAA  Academy  course  50321).  The  ATCS 
Screen  is  the  second  stage  of  a  multiple  hurdle 
selection  system;  the  first  stage  consists  of  the 
0PM  written  ATCS  aptitude  test  battery.  The 
ATCS  Screen  is  a  9-week  program  administered 
in  3  p-.ases.  First,  newly  hired  controllers 
receive  4  weeks  of  classroom,  academic 
instruction  on  nonradar  air  traffic  control  rules 
and  principles.  There  are  4  multiple-choice, 
written  examinations  corresponding  to  the  blocks 
of  instruction  provided  in  the  academic  phase  of 
the  ATCS  Screen.  ATCS  Screen  students  are 
also  required  to  draw  the  map  of  the  synthetic 
training  air  space  from  memory.  These  block 
test  and  map  scores  are  combined  into  an  overall 
academic  block  average  (NLBA).  A 
comprehensive  phase  test  (NLCPT)  is 


administered  at  the  end  of  the  academic  phase  of 
instruction.  The  block  average  counts  as  8%  of 
the  student’s  final  composite  score  in  the  Screen; 
the  comprehensive  phase  test  accounts  for  12% 
of  that  final  grade.  The  second  major  phase  of 
the  Screen  focuses  on  hands-on  application  of 
those  rules  and  principles  in  laboratory 
simulation  scenarios.  This  laboratory  phase 
requires  approximately  5  weeks.  There  are  a  total 
of  6  graded  laboratory  problems;  the  best  5 
scores  are  used  to  compute  a  student’s  final 
grade  in  the  ATCS  Screen  (AVL5).  The  grade 
on  each  laboratory  scenario  is  composed  of  a 
technical  score  and  an  instructor  assessment  of 
the  student’s  potential,  equally  weighted. 
Laboratory  scores  account  for  60%  of  the 
student’s  composite  final  score  in  the  Academy 
program.  The  final  phase  of  the  Screen  program 
consists  of  a  timed,  written  multiple-choice 
examination.  This  Controller  Skills  Te.st 
(NLCST)  accounts  for  20%  of  the  final, 
composite  ATCS  Screen  score  (NLCOMP) 
earned  by  the  student  at  the  FAA  Academy.  A 
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minimum  final  score  of  70  is  required  to  pass  the 
ATCS  Screen;  students  earning  less  than  70  fail 
the  course.  The  majority  of  failing  students  are 
terminated  from  employment  with  the  FAA. 

Two  analyses  were  conducted  for  this 
evaluation.  First,  the  pass  and  fail  rates  of  the  2 
groups  were  compared.  The  pass  rate  in  the 
ATCS  Screen  for  controllers  selected  from  the 
Airway  Science  register  (41.7%)  was 
significantly  lower  than  the  pass  rate  for  the 
sample  of  controllers  hired  by  traditional  means 
(58.7%;  Z  =  -4.25,  p  ^  .01).  As  might  be 
expected,  the  failure  rate  for  Airway  Science 
hires  (50.3%)  was  higher  than  that  of  traditional 
hires  (31.4%;  Z  =  5.15,  p  :<  .01).  The 
withdrawal  rates  for  the  2  groups  (8.0%  versus 
9.9%  for  traditional  hires)  were  not  statistically 
different  (Z  =  -0.12,  p  >  .05).  These  pass, 
fail,  and  withdrawal  rates  for  each  group  are 


presented  in  Figure  2.  Second,  the  Screen 
component  and  final  composite  scores  were 
submitted  to  a  1-way  analysis  of  variance 
(ANOVA)  by  group  (Airway  Science  versus 
traditional  hires).  The  results  of  that  analysis  are 
presented  in  Table  4.  The  academic  scores 
(NLBA  and  NLCPT)  for  the  controllers  hired 
from  the  Airway  Science  register  were 
statistically  lower  than  the  scores  for  the 
stratified,  random  sample  of  traditionally-hired 
controllers.  The  performance  of  Airway  Science 
hires  was  also  significantly  lower  in  the  graded 
laboratory  scenarios  than  for  the  traditionally- 
hired  specialists.  That  same  pattern  was  repeated 
on  the  Controller  Skills  Test.  And  as  might  be 
expected  in  view  of  the  pass  rate  differences,  the 
mean  composite  final  scores  for  controllers  hired 
under  the  ASCDP  {M  =  57.2)  was  significantly 
less  than  the  mean  composite  final  scores  (M  = 
70.7)  for  the  stratified  sample  of  controllers  hired 
from  the  traditional  competitive  register. 


Figure  2 

ATCS  Screen  outcomes  by  comparison  group 


ro 


WltMrew  Failed  Peased 


ASCDP  ^  Traditional 


General  performance  appraisals 

The  FAA  general  performance  appraisal 
system  is  design^  to  describe  specific  employee 
strengths  and  weaknesses  relative  to  pre- 


established  written  performance  standards. 
Supervisors  are  required  t».  review  an  employee’s 
position  description  and  identify  critical  job 
elements.  Each  critical  job  element  is  weighted 
with  regard  to  its  contribution  or  importance  to 
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the  position.  Specific  standards  for  "outstanding," 
"exceptional,"  "fully  successful,"  "partially 
successful,"  and  "unacceptable"  performance  on 
those  critical  job  elements  are  defined  by  the 
supervisor.  TTie  employee’s  performance  is 
appraised  relative  to  those  specific  standards  in 
the  course  of  the  evaluation.  TTiose  ratings  for 
each  critical  Job  element  are  then  weighted;  the 
weighted  scores  are  then  summed  to  create  an 
overall  composite  rating.  A  composite  score  of 
2.7  to  3.n,  with  no  critical  Job  element  rated  at 
less  than  exceptional,  earns  an  "outstanding" 
rating.  An  "exceptio-  rating  requires  a 
composite  score  of  1.0  to  2.69,  with  no  Job 
element  rated  at  less  than  fully  successful.  A 
score  of  0  to  .99,  with  no  critical  Job  element 
rated  at  less  than  partially  successful,  receives  a 
"partially  successful"  rating. 


The  employee’s  most  recent  composite, 
categorized  rating  was  extracted  from  the  CPMIS 
for  Ais  analysis.  Scores  for  121  Airway  Science 
hires  and  172  traditional  hires  were  obtained 
from  CPMIS.  These  composite  scores  were 
submitted  to  a  1-way  ANOVA  by  comparison 
group.  The  average  general  performance 
appraisal  composite  score  for  Airway  Science 
controllers  was  2.20.  The  mean  performance 
rating  score  for  the  sample  of  traditionally-hired 
specialists  was  2.08.  No  critical  Job  element  for 
either  group  received  a  less  than  "fully 
acceptable"  rating.  The  differen.es  between 
means  were  statistically  significant  (F(l,291)  = 
5.88,  p  .05).  Airway  Science  hires  appear  to 
earn  higher  ratings  on  general  performance 
appraisals  than  traditionally-hired  controllers. 


Table  4 

Results  cf  i-Way  ANOVA  of  ATCS  Screen  Scores  by  Comparison  Group 


Measure 

Group 

N 

M 

Min-Max 

SD 

F 

NLBA 

ASCDP 

304 

"8.6 

0.0  -  100.0 

34.3 

28.8“ 

Traditional 

293 

90.6 

0.0  -  100.0 

17.7 

NLCPT 

ASCDP 

304 

75.2 

0.0  -  100.0 

33.1 

28.1“ 

Traditional 

293 

86.8 

0.0  -  100.0 

17.9 

AVL5 

ASCDP 

286 

51.1 

0.0  -  92.6 

27.4 

42.7“ 

Traditional 

279 

63.8 

0.0  -  95.5 

17.4 

NLCST 

ASCDP 

286 

58.3 

0.0  -  98.0 

30.3 

56.4“ 

Traditional 

279 

74.4 

0.0  -  96.0 

19.4 

NLCOMP 

ASCDP 

286 

57.2 

0.0  -  94.4 

28.5 

47.0“ 

Traditional 

279 

70.7 

0.0  -  91.4 

16.7 

***p  :<  .001 


Acceptance  of  Technological  Change 

The  fifth  objective  of  the  ASCDP  was  the 
development  of  an  increased  acceptance  of 
technological  change  within  the  FAA’s  critical 
occupations.  The  FAA  Capital  Investment  Plan 
is  the  FAA’s  comprehensive  program  for 
modernizing  and  improving  the  National  Airspace 
System.  The  plan  objectives  are  to  meet  current 
and  future  demands  for  air  traffic  services, 
reduce  errors  and  aviation  hazards,  make  better 
use  of  personnel  and  equipment,  and  create  a 


foundation  for  future  improvements.  A  critical 
piece  of  this  plan  is  the  Advanced  Automation 
System,  a  consolidated  network  of  high-speed, 
high-efficiency  computers  and  advanced 
controller  workstations.  Controllers  will  use 
sophisticated  computer  graphics,  decision  aiding 
software,  and  procedures  to  manage  complex 
traffic.  Controller  acceptance  of  the  changes  in 
the  Job  brought  about  by  this  increased 
automation  is  important  to  the  organizational  and 
technological  success  of  the  Advanced 
Automation  System. 
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Therefore,  Clough  (1987a)  developed  5  items 
for  the  annual  Status  Survey  to  assess  attitudes 
toward  automation.  As  reported  by  Broach 
(1990),  2  scales  were  derived  from  the  5  items 
through  factor  analysis.  The  "Automation 
Effects"  scale,  based  on  3  items,  assessed  the 
degree  to  which  respondents  perceived  that  their 
effectiveness,  job  satisfaction,  and  challenge 
experienced  in  the  job,  would  change  with 
increased  automation.  Scale  values  were  as 
follows;  1  =  Definitely  negative  effect',  2  = 
Probably  negative  effect',  3  =  Neutral  or  no 
effect',  4  =  Probably  positive  effect',  5  = 
Definitely  positive  effect.  The  "Automation 
Changes"  scale,  derived  from  2  survey  items, 
assessed  the  degree  that  automation  had  changed 
or  was  expect^  to  change  the  job  itself  in  the 
respondent’s  view.  Scale  values  were;  1  =  No 
change',  2  =  Little  change',  3  =  Some  change-,  4 
=  Much  change',  5  =  Extensive  change.  These 
ratings  were  submitted  to  1-way  ANOVA 


by  comparison  group  for  each  year  of  the  Status 
Survey,  as  presented  in  Table  5. 

Controllers  hired  from  the  Airway  Science 
register  completing  the  first  status  survey  (Year 
1;  A'  =  74)  indicated  that  automation  would 
probably  have  a  positive  effect  on  their  job 
effectiveness,  satisfaction,  and  degree  of 

challenge  experienced  {M  =  3.76).  As  shown 
in  Table  8,  specialists  hired  by  the  traditional 
process  who  completed  the  first  status  sur\'ey  (N 
=98)  also  indicated  that  automation  would 
probably  have  a  positive  effect  on  their  job 
effectiveness,  satisfaction,  and  experienced 

degree  of  challenge  {M  =  3.49).  Airway 
Science  hires  were  more  positive  than  traditional 
hires  in  this  perception  (F(l,170)  =  5.67,  p  ^ 
.05).  However,  the  practical  importance  of  a 
1/4-point  difference  in  mean  ratings  must  be 
considered,  given  that  as  sample  sizes  increase, 
smaller  and  smaller  differences  become 


Table  5 

Acceptance  of  Technological  Change  by  Comparison  Group 


Year 

Group 

N 

M 

Min-Max 

SD 

F 

Perceived  Effect  of  Automation  on  Job 

1 

ASCDP 

74 

3.76 

2  -  5 

0.72 

5.67’ 

Traditional 

98 

3.49 

1  -  5 

0.74 

2 

ASCDP 

19 

3.68 

2  -  5 

0.89 

0.68 

Traditional 

18 

3.89 

3-5 

0.58 

3 

ASCDP 

7 

3.00 

3  -5 

0.82 

3.56 

Traditional 

11 

3.73 

2  -  4 

0.79 

Perceived  Change  in  the  Job  Due  to  Automation 

1 

ASCDP 

98 

2.76 

1  -  5 

0.83 

1.14 

Traditional 

76 

2.92 

2  -  5 

o.% 

2 

ASCDP 

18 

3.17 

2-5 

1.20 

2.49 

Traditional 

19 

3.74 

1  -5 

0.99 

3 

ASCDP 

7 

3.43 

2-5 

1.13 

0.18 

Traditional 

11 

3.64 

2  -  5 

0.92 

*  F  ,<  .05 
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"statistically  significant."  The  mean  Automation 
Effects  rating  (Af  =  3.68)  for  the  19  Airway 
Science  controllers  that  completed  the  second 
status  survey  (year  2)  was  not  statistically 
different  from  the  rating  given  by  18  traditional 
hires  (M  =  3.89;  F(l,35)  =  0.68,  p  >  .05). 
Similarly,  the  mean  Automation  Effects  rating 
for  just  1 1  Airway  Science  controllers  completing 
the  third  Status  Survey  (Year  3,  M  =  3.72)  was 
not  statistically  different  from  that  given  by  just 
7  traditional  hires  (M  =  3.00;  F(l,16)  =  3.56, 
p  >  .05).  Overall,  it  appears  that  there  were  no 
practical  differences  between  the  2  groups  in 
their  perceptions  of  the  effects  of  automation  on 
their  job  effectiveness,  satisfaction,  and  degree  of 
challenge  experienced  in  the  job. 

A  similar  pattern  in  the  perceptions  of 
changes  in  the  job  due  to  automation  was 
observed  between  groups.  Controllers  hired 
from  the  Airway  Science  register  responding  to 
the  first  status  survey  (Year  \  \  N  =  74)  perceive 
or  expect  some  degree  of  change  in  their  jobs 
due  to  automation  (M  -  2.76).  Specialists  hired 
from  the  traditional  registers  completing  the  first 
status  survey  (A  =  98)  also  perceive  or  expect 
that  some  change  in  their  job  due  to  automation 
(M  =  2.92).  The  difference  in  means  was  not 
statistically  significant  (F(l,172)  =  1.14,  p  > 
.05).  The  mean  Automation  Changes  ratings  for 
the  18  Airway  Science  hires  completing  the 
second  status  survey  (Year  2;  M  =  3.17)  was 
not  statistically  different  from  the  rating  given  by 
the  19  traditional  hires  in  their  second  status 


survey  (Af  =  3.74;  F(l,35)  =  2.49,  p  >  .05). 
As  presented  in  Table  6,  there  were  no 
differences  in  mean  ratings  of  Automation 
Changes  by  group  for  those  controllers  who 
completed  die  third  annual  status  survey.  Both 
groups  in  the  third  survey  indicated  expecting  or 
perceiving  "much"  change  in  their  jobs  due  to 
automation.  The  increase  across  years  in  the 
degree  of  change  expected  or  perceived  as  a 
consequence  of  automation  might  be  explained  by 
increasing  awareness  of  the  FAA  Advanced 
Automation  System  as  the  controllers  move 
through  training,  coincident  with  increased 
publicity  for  and  attention  to  that  automation 
system.  Overall,  however,  the  controllers  hired 
from  the  Airway  Science  registers  appeared  about 
as  receptive  to  automation  as  specialists  hired 
from  the  traditional  registers. 

Managerial  Potential 

The  sixth  objective  of  the  ASCDP  was  to 
recruit  individuals  with  managerial  potential. 
Clough  (1987a)  designed  the  5-item  "Index  of 
Managerial  Potential"  (IMP)  to  assess  the  degree 
to  which  participants  demonstrated  skills  linked 
to  effective  management.  Supervisors  of 
participating  employees  completed  the  IMP  once 
a  year.  Analyses  of  the  IMP  by  Broach  (1990) 
indicated  that  a  single  dimension  captured  70% 
of  the  variance  across  IMP  dimensions. 
Therefore,  IMP  scales  were  collapsed  within  a 
year  of  data  collection  to  create  an  overall, 
summary  index  of  managerial  potential  for 


Table  6 

Managerial  Potential  by  Comparison  Group 


Year 

Group 

N 

M 

Min  -  Max 

SD 

F 

1 

ASCDP 

90 

3.39 

2  -  5 

0.77 

0.14 

Traditional 

140 

3.43 

1  -  5 

0.80 

2 

ASCDP 

14 

3.57 

2  -  5 

0.76 

4.81' 

Traditional 

17 

2.94 

1  -5 

0.83 

3 

ASCDP 

3 

3.33 

3-4 

0.58 

0.50 

Traditional 

3 

2.61 

1  -4 

1.53 

♦  p  .<  .05 
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analysis.  These  single-scale  scores  for  each  year 
were  submitted  to  one-way  ANOVA  by 
comparison  group;  the  results  of  the  analysis  are 
presented  in  Table  6. 

Managerial  potential  was  indexed  on  the 
following  scale:  1  =  Definitely  does  not  have 
potential]  2  =  Probably  does  not  have  potential] 
3  =  Uncertain  about  potential]  4  =  Probably 
has  potential]  and  5  =  Definitely  has  potential. 
There  were  no  significant  differences  between 
groups  after  1  year  of  employment  in  perceived 
managerial  potential  (ASCDP  M  =  3.39; 
Traditional  M  =  3.43,  F(l,228)  =  0.14,  ns). 
However,  these  results  might  be  explained  by 
considering  that  field  supervisors  had  relatively 
little  exposure  to  the  new  employee  after  1  year 
of  employment,  as  a  large  proportion  of  that 
employee’s  time  was  spent  at  the  FAA  Academy 
and  in  facility  classrooms  with  little  direct 
interaction  with  the  supervisor.  Comments 
received  from  field  supervisors  suggested  that 
they  felt  uncomfortable  trying  to  assess  a  new 
employee’s  managerial  potential.  However, 
supervisors  rated  the  managerial  potential  of 
Airway  Science  hires  after  2  years  of 
employment  more  highly  {M  =  3.57)  than  that  of 
traditional  hires  (M  =  l94;  F(],29)  =  4.81,  p 
^  .05).  The  number  of  supervisors  completing 


the  assessment  of  employee  managerial  potential 
after  3  years  of  employment  was  too  small  to 
support  meaningful  statistical  analysis.  The  data 
suggest  that  there  might  be  differences  in  the 
perceived  managerial  potential  of  the  2  groups; 
however,  the  low  rates  of  return  in  the  second 
and  third  years  of  data  collection  preclude  valid 
conclusions  from  being  drawn  from  the  data. 

Human  Relations  Skills 

The  seventh  objective  of  the  ASCDP  was  to 
attract  individuals  with  greater  awareness  of  and 
skills  in  maintaining  positive  human  relations. 
This  objective  was  operationally  represented  by 
Clough  (1987a)  as  a  single  facet  within  the  Index 
of  Managerial  Potential.  Supervisors  of  the 
participants  in  the  demonstration  project  rated 
each  person  on  their  "interpersonal  skills"  each 
year.  Supervisors  responded  to  the  question, 
"How  would  you  describe  this  individual’s 
interpersonal  skills"  on  the  following  5-point. 
Likert-type  scale:  1  =  Poor.  2  =  Somewhat 
below  average]  3  =  Average]  4  =  Somewhat 
above  average]  5  =  Excellent.  These  ratings  of 
interpersonal  skills  after  1  year  (Year  1),  2  years 
(Year  2),  and  3  years  (Year  3)  of  employment 
were  submitted  to  1-way  ANOVA  by  comparison 
group.  The  results  of  that  analysis  are  presented 
in  Table  7. 


Table  7 

Rated  Human  Relations  Skills  by  Comparison  Group 


Year 

Group 

N 

M 

Min  -  Max 

SD 

F 

1 

ASCDP 

90 

3.49 

2  -  5 

0.81 

0.25 

Traditional 

139 

3.55 

1  -  5 

0.89 

2 

ASCDP 

14 

3.29 

2  -  4 

0.61 

1.22 

Traditional 

17 

3.00 

1  -  5 

0.79 

3 

ASCDP 

3 

4.00 

2  -  4 

0.00 

4.00 

Traditional 

3 

3.33 

1  -  4 

0.58 

*  p  :<  .05 


There  were  no  differences  in  the  rated  human 
relations  or  interpersonal  skills  between  the 
Airway  Science  hires  (A/  =  3.49)  and  controllers 
hired  by  traditional  means  {M  =  3.55,  F(l,227) 
=  0.25,  p  >  .05)  after  1  year  of  employment. 


Nor  were  there  any  significant  differences 
between  groups  after  2  or  3  years  of 
employment,  although  the  small  sample  sizes 
preclude  reliable  statistical  inferences.  Overall, 
the  data  show  that  the  perceived  human  relations 
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skills  of  controllers  hired  from  the  Airway 
Science  register  were  not  different  from  that  of 
specialists  hired  via  traditional  means. 

Female  and  Minority  Representation 

The  next  objective  of  the  ASCDP  was  to 
increase  the  representation  of  women  and 
minorities  within  the  FAA  technical  occupations. 
This  objective  was  operationally  represented  as 
the  relative  rates  of  organizational  entry  and 
attrition  for  women  and  minorities. 
Organizational  attrition  was  defined  as  having  left 
the  service  of  the  FAA  as  of  the  end  of  calendar 
year  1990,  as  reflected  in  CPMIS  nature-of- 
action  codes.  Consider  first,  new  hires  entering 
onto  duty  with  the  agency  from  each  group.  The 
percentage  of  minority  new  hires  in  the  control 
group  (6. 1  %)  was  not  significantly  different  from 
that  of  minority  new  hires  in  the  Airway  Science 


group  (10.3%,  Z  =  \.9\,  p  >  .05).  The 
percentage  of  women  in  the  control  group  sample 
(18.6%)  entering  the  agency’s  employ  was  not 
significantly  different  from  that  for  the  Airway 
Science  group  (13.8%;  Z  =  -1.63,  p  >  .05). 
The  attrition  and  retention  rates  for  the 
comparison  samples  of  controllers  are  presented 
by  minority  status  and  sex  of  participant  in  Table 
8.  First,  the  retention  rates  for  minority  and 
majority  participants  within  each  sample  (e.g., 
ASCDP  hires)  were  not  statistically  different 
(minority  v.  majority  Airway  Science,  Z  =  0.78, 
p  >  .05;  minority  v.  majority  Traditional  hires, 
Z  =  0.32,  p  >  .05).  Second,  the  organizational 
retention  rate  of  minority  Airway  Science  hires 
was  not  significantly  different  from  that  of 
minority  Traditional  hires  (Z  =  0.64,  p  >  .05). 
Third,  the  retention  rates  for  men  and  women 
within  each  comparison  sample  (e.g..  Traditional 
hires)  were  not  significantly  different.  For 


Table  8 

Organizational  Retention  Rates  by  Comparison  Group,  Sex,  and  Minority  Status 


Group 

Sample 

Attrited 

Retained 

By  Minority  Status  of  Participant 

Minority 

ASCDP 

13 

19 

(40.6%) 

(59.4%) 

Traditional 

6 

13 

(46.2%) 

(53.8%) 

Majority 

ASCDP 

111 

120 

(48.1%) 

(51.9%) 

Traditional 

102 

188 

(35.2%) 

(64.8%) 

By 

Sex  of  Participant 

Female 

ASCDP 

22 

21 

(51.2%) 

(48.8%) 

Traditional 

24 

34 

(41.4%) 

(58.6%) 

Male 

ASCDP 

108 

120 

(47.4%) 

(52.6%) 

Traditional 

86 

168 

(33.9%) 

(66.1%) 
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example,  48.8%  of  the  women  hired  from  the 
Airway  Science  register  were  retained  by  the 
FA  A,  while  52.6%  of  the  men  hired  from 
thatregister  remained  with  the  agency  (Z  =  0.45, 
p  >  .05).  Similarly,  the  retention  rate  of  58.6% 
for  traditionally-hired  women  hired  was  not 
significantly  less  than  that  of  traditionally-hired 
men  (66.1%;  Z  =  -1.09,  p  >  .05).  Finally,  the 
retention  rate  for  women  hired  from  the  Airway 


Science  register  (48.8%)  was  not  significantly 
less  than  the  rate  for  women  hired  via  the 
traditional  registers  (58.6%;  Z  =  -0.98,  p  > 
.05). 

FAA  Employee  Perceptions 

Employee  perceptions  ot  the  FAA,  reflected 
in  measures  of  job  satisfaction,  continue  to  be  an 


Table  9 

Facet  Job  Satisfaction  by  Comparison  Group 


Year 

Group 

N 

M 

Min  -  Max 

SD 

F 

Personal  Satisfaction  with  Job 

1 

ASCDP 

76 

4.08 

1  -  5 

0.73 

2.09 

Traditional 

98 

3.94 

1  -  5 

0.55 

2 

ASCDP 

19 

4.00 

2  -  5 

0.75 

0.94 

Traditional 

18 

3.78 

2  -  5 

0.65 

3 

ASCDP 

11 

4.00 

2  -  5 

0.77 

1.63 

Traditional 

7 

3.57 

3  -  4 

0.53 

Satisfaction  with  Co-workers 

1 

ASCDP 

76 

3.80 

1  -  5 

0.78 

0.09 

Traditional 

98 

3.84 

2  -  5 

0.68 

2 

ASCDP 

19 

3.84 

1  -  5 

1.01 

0.05 

Traditional 

18 

3.78 

2  -  5 

0.65 

3 

ASCDP 

11 

3.27 

1  -  4 

1.01 

0.12 

Traditional 

7 

3.43 

2  -  4 

0.79 

Satisfaction  with 

Supervisor 

1 

ASCDP 

76 

3.78 

1  -  5 

0.81 

0.88 

Traditional 

98 

3.66 

1  -  5 

0.77 

2 

ASCDP 

19 

3.84 

1  -  5 

1.07 

0.86 

Traditional 

18 

3.56 

2  -  5 

0.78 

3 

ASCDP 

11 

2.73 

1  -  4 

1.19 

2.12 

Traditional 

7 

3.43 

3  -  4 

0.53 

(Table  9  continues) 
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(Table  9  continued) 


Year  Group  N  M  Min  -  Max  SD  F 


Satisfaction  with  Degree  of  Autonomy 


1 

ASCDP 

76 

3.14 

2-5 

0.69 

2.09 

Traditional 

98 

2.99 

2  -  5 

0.71 

2 

ASCDP 

19 

3.16 

2  -  5 

0.76 

0.39 

Traditional 

18 

3.00 

2  -  5 

0.77 

3 

ASCDP 

11 

3.27 

2  -  4 

0.65 

0.62 

Traditional 

7 

2.71 

2  -  4 

1.11 

Satisfaction  with  Pay 


1 

ASCDP 

76 

3.42 

1  -  5 

0.96 

0.82 

Traditional 

98 

3.55 

1  -  5 

0.92 

2 

ASCDP 

14 

3.14 

1  -  5 

1.03 

0.01 

Traditional 

11 

3.18 

1  -5 

0.87 

3 

ASCDP 

11 

2.82 

1  -  5 

1.60 

0.02 

Traditional 

7 

2.71 

1  -  4 

l.ll 

Satisfaction  with  Performance  Evaluation/ Appraisal  System 


1 

ASCDP 

76 

3.07 

1  -  5 

0.77 

0.44 

Traditional 

98 

2.99 

1  -  4 

0.74 

2 

ASCDP 

19 

2.89 

1  -  4 

0.88 

0.33 

Traditional 

18 

2.72 

1  -  4 

0.96 

3 

ASCDP 

11 

2.55 

1  -  3 

0.69 

0.01 

Traditional 

7 

2.57 

2  -  3 

0.53 

issue  of  organizational  importance,  as  reflected  in 
the  biennial  FA  A  job  satisfaction  surveys  (Myers, 
Schroeder,  Van  Deventer,  &  Collins,  1988; 
Schroeder,  Collins,  &  Dollar,  1984;  Schroeder, 
Thomas,  Weltin,  Van  Deventer,  Collins,  Dollar, 
&  Ritchie,  1986).  Twenty-six  items,  developed 
in  collaboration  with  0PM  researchers,  were 
incorporated  into  the  annual  status  surveys  to 
assess  various  facets  of  Job  satisfaction.  Broach 
(1990)  used  factor  analysis  to  identify  6 
underlying  dimensions:  (a)  personal  satisfaction; 
(b)  satisfaction  with  co-workers;  (c)  satisfaction 


with  supervisors;  (d)  satisfaction  with  degree  of 
autonomy  afforded  the  participant;  (e)  satisfaction 
with  pay;  and  (0  satisfaction  with  the 
performance  evaluation/appraisal  process.  Facet 
job  satisfaction  was  indexed  on  the  following  5- 
point,  Likert-type  scale;  1  =  Very  dissatisfied', 
2  =  Dissatisfied',  3  =  Neutral',  4  =  Satisfied', 
and  5  =  Very  satisfied.  Scores  on  each  facet 
were  submitted  to  1-way  ANOVA  by  comparison 
group  for  each  year.  The  results  of  that  analysis 
are  presented  in  Table  9. 


The  personal  satisfaction  facet  included  items 
assessing  overall  personal  satisfaction  with  the 
job,  performing  meaningful  work,  general 


satisfaction,  and  how  much  the  employee  liked 
working  for  the  FAA.  As  presented  in  Table  9, 
there  were  no  significant  differences  between 
Airway  Science  controllers  and  traditionally- 


Table  10 

Summary  Evaluation  o/ASCDP  Relative  to  Program  Objectives 


Project  Objective 

Evaluation  Criteria 

Outcome 

Objective  2 A 

To  assess  the  unique  impact  of 

Aviation-related  education  completed 

-1- 

a  tailor-made  curriculum  on 

Aviation-related  work  experience,  etc. 

+ 

one’s  interest  in  pursuing  an 
aviation-related  career. 

Rated  importance  of  aviation-related  career 

-1- 

Objective  2B 

To  assess  the  unique  impact  of 

Occupational  attrition 

- 

a  tailor-made  curriculum  on 
occupational  and  organizational 
attrition 

Organizational  attrition 

0 

Objective  3 

To  improve  the  technical 

Occupational  training  performance 

- 

competence  of  the  FAA 
workforce 

Formal  performance  appraisal  results 

-1- 

Objective  4 

To  develop  within  the  FAA  an 

Effect  of  automation  on  job  perceptions 

0 

increased  acceptance  of 
technological  change 

Perceived  changes  in  job  from  automation 

0 

Objective  5 

To  recruit  individuals  with 
managerial  potential 

Rated  managerial  potential 

0 

Objective  6 

To  attract  individuals  with 

Rated  interpersonal  skills 

0 

greater  awareness  of  and  skills 
in  maintaining  positive  human 
relations 

Objective  7 

To  increase  female  and 
minority  representation 

Proportions  of  females/minorities  retained 

0 

Objective  8 

To  improve  employee  per- 

Degree  that  FAA  meets  employee  work  needs 

0 

ceptions  of  the  FAA 

Job  satisfaction 

0 

Notes:  Following  symbols  are  used  in  the  "Outcome"  column;  -  =  ASCDP  lower  than  traditional 
hires;  0  =  No  difference  between  groups;  +  =  ASCDP  higher  than  traditional  hires  on  criteria 
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hired  controllers  in  the  degree  of  personal  job 
satisfaction  after  1,  2,  or  3  years  of  employment. 
Generally,  participating  employees  were 
"Satisfied."  However,  as  before,  the  data  for  the 
second  and  third  year  must  be  interpreted 
cautiously  due  to  the  low  rate  of  return  of 
surveys.  The  facet  scale  assessing  employee 
satisfaction  with  co-workers  was  derived  from  4 
items  (Broach,  1990).  A  sample  item  is,  "My 
group  works  well  together."  There  were  no 
significant  differences  between  groups  in  the 
degree  to  which  they  were  satisfied  with  their  co¬ 
workers.  Both  Airway  Science  and  traditionally- 
hired  controllers  were  "Satisfied"  with  their  co¬ 
workers.  Satisfaction  with  one’s  supervisor  was 
the  third  facet  scale  evaluated  in  this  study,  also 
based  on  4  items  from  the  annual  Status  Survey. 
"My  supervisor  helps  me  solve  work-related 
problems"  is  an  example  item  from  this  facet. 
As  with  the  previous  facets  of  Job  satisfaction, 
there  were  no  significant  differences  between 
Airway  Science  and  traditionally-hired  controllers 
in  the  degree  of  their  satisfaction  with  their 
supervisors  after  1,  2,  or  3  years  of  employment. 
Overall,  both  groups  were  "Satisfied"  with  their 
supervisors. 

The  next  job  satisfaction  facet  to  be  analyzed 
was  employee  satisfaction  with  the  degree  of 
autonomy  offered  in  the  job  of  air  traffic  control 
specialist.  An  example  from  this  set  of  5  items 
is,  "I  have  control  over  how  I  spend  my  time 
working."  As  presented  in  Table  9,  the  same 
pattern  as  before  was  found,  with  no  significant 
differences  between  groups  across  years  of 
employment.  While  both  groups  had  been 
"Satisfied"  on  the  preceding  facet  scales, 
controllers  appeared  to  be  neither  satisfied  nor 
dissatisfied  regarding  the  degree  of  autonomy 
experienced  on  the  job.  Satisfaction  with  pay 
was  assessed  by  2  items.  An  example  item  is, 
"  Considering  my  skills  and  the  effort  I  put  into 
my  work,  I  am  satisfied  with  my  pay."  There 
were  no  significant  differences  between  groups. 
The  controllers  appeared  to  be  neutral,  though 
there  appeared  to  be  a  trend  across  the  years 
toward  greater  dissatisfaction  with  pay.  The 
final  job  satisfaction  facet  scale  was  derived  from 
3  items  reflecting  satisfaction  with  the 
performance  evaluation/appraisal  system.  A 
sample  item  is,  "My  performance  rating 
represents  a  fair  and  accurate  picture  of  my 
actual  performance."  As  with  pay  satisfaction, 
there  were  no  significant  differences  between 
groups  within  each  year  of  employment,  with  the 
groups  being  neither  satisfied  nor  dissatisfied. 


However,  the  ratings  for  this  job  satisfaction 
appeared  to  be  somewhat  lower,  overall,  than  for 
other  facets  of  the  job.  Overall,  the  job 
satisfaction  of  controllers  hired  from  the  Airway 
Science  register  appeared  to  be  indistinguishable 
from  that  of  controllers  hired  via  traditional 
registers. 

CONCLUSIONS 

The  pattern  of  evaluation  results  is 
summarize  in  Table  10.  Airway  Science  hires 
appeared  to  have  more  interest  in  pursuing  an 
aviation-related  career,  as  evidenced  by  the 
amount  of  aviation  education  completed,  aviation 
work  experience  reported,  and  rated  importance 
of  a  career  in  aviation.  The  occupational  attrition 
rate  for  airway  Science  hires  was  greater  than  the 
attrition  rate  for  traditionally  hired  controllers; 
overall  organizational  loss  rates  were  the  same 
for  the  two  groups.  Airway  Science  hires 
performed  as  well  as  traditional  hired  in  initial 
occupational  training,  but  received  higher  formal 
performance  ratings  on  the  job.  There  were  no 
differences  between  the  groups  with  respect  to 
attitudes  toward  automation,  rated  managerial 
potential,  interpersonal  skills,  minority  and 
female  retention  rates,  and  job  satisfaction. 
Overall,  the  data  presented  in  this  summative 
evaluation  indicate  that  the  cadre  of  individuals 
produced  by  the  demonstration  project’s  hiring 
strategy  are  indistinguishable  from  more 
traditionally  hired  persons.  The  fundamental 
design  of  this  evaluation,  however,  precludes  a 
ca.susal  interpretation  of  effects,  or  conversely, 
lack  of  enhancing  effects,  from  the  Airway 
Science  curriculum. 
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